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FOREWORD 



The dynamic process of educating American youth depends on continued 
community support and feedback of information to the administrative 
leadership. There is a need to develop effective methods for determining 
attitudes and opinions on important aspects of educational systems under 
development . 

It was the purpose of the study described here to gather additional 
information about Performance Indicators in Education, a New York State 
Education Department project designed to improve the ability of school 
districts to evaluate their performance. This report describes the 
results of a study of superintendents' perceptions of the Performance 
Indicators project as well as the experimental application of a telephone 
survey technique to collect data for the study. 

The telephone survey technique represents a rapid means of securing 
reliable and ti;ae!y data. It has a major advantage over other survey 
methods— speed. Th-?. potential of the technique was illustrated in the 
study reported herein. It demonstrated that reactions of school personnel 
to new developments in education can be ascertained and repackaged for 
decision making in an economical, efficient, and timely manner. 

This twofold study also represents a collaborative effort by the 
Bureau of School Programs Evaluation and the Bureau of Statistical Services, 
David J. Irvine, chief of the former bureau, directs the Performance 
Indicators in Education program. Lee R. Wolfe, chief of the latter 
bureau, developed the methodology of the telephone survey technique. 



iii 
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supervised the data collection, and contributed substantial portions to 
the manuscript. Guy D. Spath, associate in edtication research, and 
Greg M. Lepak, statiiitician, monitored the calling, compiled the data, 
and prepared tne veport. Edith Tracy, Mable Purello, and Joanne Havlik 
carried out the bulk of the calling. 
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CHAPTER I 



INTRODUCTION 



The Performance Indicators in Education program is intended to 
provide measures of the impact school districts have on the achievement 
of their pupils. After an extensive period of development by the Bureau 
of School Programs Evaluation of the New York State Education Department, 
reports were prepared for 628 school districts in New York State. They 
were distributed at regional meetings to school superintendents or their 
representatives, or by mail to those districts not represented at a 
meeting. The importance of shaping the program to meet the needs of 
local educators was recognized since the use of any information system 
depends upon the intended users' perceptions of the validity, utility, 
and applicability of the program. A way was sought to obtain more 
systematic information about the reception and use of the Perforniance 
Indicators reports. After consultation with the Bureau of Statistical 
Services, a telephone survey of a random sample of chief school admini- 
strators seemed to be a feasible method of obtaining the desired 
information. Staff members of the two bureaus clarified objectives, 
developed the methodology, and carried out the survey. 

Initial Considerations and Procedures 

The survey had two purposes: 1) to collect information on the 
Performance Indicators program; and 2) to develop the telephone survey 
as a technique for rapid data collection and analysis. 

Information was needed about reactions to the Performance Indicators 
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program to report progress and to make decisions about future directions 
to be taken by the Bureau, Perceptions of chief school administrators 
were sought on the validity and utility of the performance reports, how 
they were used by districts, needed improvements, and additional related 
services which might be desirable, 

A previous telephone survey was conducted in Greece, New York, to 
obtain data about the attitude of citizens toward Project Redesign in 
that district.^ Experience there indicated that information can be 
collected and tabulated very rapidly, even without high level technical 
devices (e.g., computer) and at a minimal cost. With the installation of 
a statewide telephone tieline for the State Education Department, the 
approach seemed feasible. Considering the paper-shuffling morass faced 
by chief school administrators as well as Department personnel, this 
technique appeared refreshing, to say the least. With this background, 
the two Bureaus agreed to a cooperative project to obtain feedback on 
Performance Indicators as well as to investigate further the use of 
telephone surveys by the State Education Department. 

After examining the Greece survey model, modifications were made to 
establish a data collection proc^idure which could be applied to diverse 
educational situations. The first tasks undertaken were to: 

1. Establish the existence of a general purpose for which the 
survey was to be taken. 

2. Translate this general purpose into a series of specific 
objectives or requirements for information. 

A major question to be ansv/ered was whether the districts were aware of 

iLee R. Wolfe. "Greece Central School District's Redesign Awareness 
Telephone Survey" (unpublished). Albany, N.Y.: Bureau of Statistical 
Services, State Education Department. July 1972. 
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Department efforts to develop the Performance Indicators systei', and dis- 
seminate reports. This central question provided the theme tor the 
questionnaire. Since, in both the Greece and the Performnce Indicators 
surveys, the feasibility of using the telephone survey technique was 
also under study, the questionnaires were deliberately restricted in 
their complexity so as not to confound evalu.^tiou of the method. 

A conceptual model had been developed for the Greece survey and is 
presented in figure 1. 
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Figure 1. Conceptual Model 

The figure shows how a decision maker (the manager of the Performance 
Indicators program, for example) can seek expert advice (the staff of the 
Bureau of Statistical Services) to obtain needed information. The callers 
collect data from relevant populations (a sample of chief school admini- 
strators) and transmit their summaries to a compiler. The compiler 
repackages the data and transmits it in turn to the decision maker. The 
decision maker can use the information to 1) report progress to higher 
level decision makers; 2) make appropriate changes in the program; and 
3) inform school districts. 

The conceptual model served as a basis for developing and carrying 
out the Performance Indicators in Education telephone survey. 
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CHAPTER II 



INSTRUMENTS AND PROCEDURES 

Before the Performance Indicators survey could be executed, the 
technical details of selecting a sample, developing a questionnaire, 
anticipating oral responses, and planning ways of repackaging responses 
for analysis had to be worked out. 

A number of sources of possible bias exis*:: for any survey which 
employs the interview as a means of collecting data. Some common sources 
of bias are: 

1. Backgrounds of caller and respondent, which are the 
sources of the perceptions, attitudes, motivations, and 
expectations they bring to the interview. 

2. Psychological factors which can be triggered by the: 

a. content of the survey 

b. initial perceptions derived from visual-auditory 
attributes 

c. behavior of the caller and respondent. 

3. Behavioral factors such as: 
a* redirected questions 

b. probes (both sides) for additional information 

c. decisions made when recording responses 

d. methods of motivating and eliciting responses. 

The telephone survey can reduce bias if it is well planned, if the 
questions are closed and simple, and if the callers are well trained. In 
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addition, the "blind" interview reduces bias due to individual 
characteristics. The data presented here are felt to be relatively free 
of systematic bias from the sources mentioned above. 

Development of instruments. The survey required the creation of 
three instruments: 

1. Callers' Instructions and Guide (appendix A). 

2. Call Record Sheet (appendix B). 

3. Caller's Evaluation (appendix C). 

In developing the instruments; the following factors were considered: 

1. Interview objectives. 

2. Respondents' level of information on the interview topic. 

3. The degree of structure which would characterize the 
respondents' opinions on the topic. 

4. Ease with which the material could be communicated by 
the caller. 

5. Methods to be used in the analysis of the interview data. 
The objectives the survey was to serve for the Performance Indicators 

program were: 

1^ To determine if and how districts received the Performance 
Indicators report. 

2. Determine the extent to which chief school administrators 
are disseminating and/or utilizing the report. 

3. To determine the extent to which the report agrees with 
other information about each district. 

4. To determine how the program can be improved. 

5. To determine the desirability of distributing a workbook 
so that districts can develop their own reports. 

12 
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6* To determine the general attitude of chief school admini- 
strators toward the program. 

7. To determine relationships among the above (e.g., did 
districts which attended a meeting disseminate the 
report to more groups than did districts which received 
the report by mail?) 
The objectives the survey was to serve for validating the technique 

were: 

1. To enable the collection of accurate and reliable 
attitude and opinion data. 

2. To achieve high speed (measured in hours) data collection. 

3. To achieve high speed data aggregation. 

4. To attain an acceptable response rate (greater than 757o). 

5. To develop an operation which can be conducted by non- 
technical personnel . 

6. To achieve these objectives at low cost. 

One of the most difficult aspects of survey research is the aggre- 
gation of the data it generates. This problem is common to all survey 
techniques and generally results in delays in repackaging data for 
decision making. Having callers fill out a questionnaire for each 
respondent would create the same tasks in compiling the data that are 
implicit in the mailed survey and was therefore rejected as a procedure. 
To overcome these difficulties, a special form, the Call Record Sheet, 
was devised (appendix B) . It contained, on a single line, provision for 
recording the responses to all questions risked of a given respondent as 
well as unforeseen responses. 



The major obstacle to overcome in developing the Call Record Sheet 
was to partition the universe of possible responses into separate 
categories. These categories had to be meaningful and representative 
of the respondents' major idea or response to the question. The caller 
had to listen to the response and assign it to one of the predetermined 
categories during the interview. Since the questions were simple and 
direct, this did not prove to be difficult for the callers. In this way, 
the Call Record Sheet became the vehicle for collecting, editing, and 
aggregating the verbal input into categories as each interview was in 
progress rather than after all data are received, as is necessary with 
mailed surveys. The caller simply added the columns on the Call Record 
Sheet by region and relayed this to the compiler who used a Data Sheet 
for collating all data. Thus, all data were summarized shortly after all 
calls had been made. 

A final instrument was created in order to obtain evaluative data 
from the callers. It was the Callers Evaluation form (appendix C). 
CalJ irs were asked to give their impressions about the respondents' 
attitudes toward the survey, to record unusual responses, and to report 
their suggestions for improving the process. 

The three instruments constituced a complete data collection system 
with an evaluation component. 

Sample se lection . A number of methods exist for selecting a statis- 
tically defensible sampling plan. The fcllov;ing technique was used in 
this study. Statistical tables^ were consulted. It was determined that 



^Table J was consulted in^'Sampling and Statistics Handbook for 
Surveys in Education,'* Research Division of the National Education 
Association, Washington, D.C,, 1965. 
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a sample of 144 districts was necessary to adequately represent the State 
Districts were randomly sampled on a regional basis to insure representa- 
tiveness. While sampling errors exist in all sampling plans, randomization 
is asstimed to minimize the effects of error. There were no known sources 
of sample bias, and it was assumed that the sample conformed to the 
population. For this application it was determined that the sample would 
be satisfactory for purposes of analysis and in terms of the judgments 
to be made upon the basis of the data. The sample was apportioned 
randomly to the four callers to further eliminate the caller bias. 

Execution of the Survey . It was estimated that four callers could 
survey the sample in one day. Four interviewers then were selected from 
the Bureau of Statistical Services staff. Members of the Bureau of 
School Programs Evaluation staff, who were familiar with the program, 
were purposely excluded from the survey to avoid possible biasing of the 
r^cults. Further, the purpose of this effort was to collect rather than 
to disseminate information; respondents would be less likely to try to 
engage in discussions of the program if they were interviewed by someone 
outside the program. To assure at least a familiarity with Performance 
Indicators so that telephone conversations would flow more smoothly, each 
dialler was asked to read the introductory pages (pp. ii-v and pp. 1-6) of 
Performance Indicators in Education: Local District Results— 1972 Report 
(Bureau of School Programs Evaluation, September 1972). 

Materials were distributed to interviewers and a short briefing was 
hell at 9 a.m., Monday, December 18. By 9:30, chief school admini- 
strators were being called on the State tieline. The four callers 
finished the survey on Tuesday, December 19. A debriefing was held and 
the data were transmitted for compiling. The results were summarized 
and analyzed as presented below. 1 5 
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CHAPTER III 



FINDINGS 



Performance Indicators reports were distributed to 628 districts. 
A representative sample of 144 die::rictP was selected to be surveyed. 
The chief school officer, or someone he designated, was successfully 
contacted in 140 districts. This represents a much higher return than 
reported for most questionnaires. Of the 140 districts contacted, 109 
indicated familiarity with the report. Greatest familiarity with the 
reports was recorded for the Long Island and Rockland-Westchester regions 
(each 100%). Results are presented in table 1. 



TABLE I 
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INDICATED FAMILIARITY WITH THE PERFORMANCE INDICATORS REPORT 
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Of the 140 districts contGCted, 92 received their reports at 
regional meetings and 48 received them by mail. To determine the 
relationship between the methods by which the report was received and 
familiarity with it, a Chi square test was made of the districts in the 
sample of 140. 



TABLE 2 

RELATIONSHIP BETWEEN HOW REPORT WAS 
RECEIVED AND REPORTED FAMILIARITY OR UNFAMILIARITY 





How Received 




Meeting 


Mail 


Familiar 


66 


43 


Unfamiliar 


26 


5 




140 Total 



A X'^ value of 4.79 was obtained which is significant at the .05 
level. Dissemination by mail seems to be more effective than meetings 
to guarantee that the superintendent is familiar with such a report. 



Of the contacted districts which received a report, 78.9 percent 
subsequently presented the report to their administrative/supervisory staff; 
33.9 percent presented the report to the teaching staff; and 33.0 percent 
presented it to the school board. 

The percents of districts not presenting the report to administrative/ 
supervisory staff, the teaching staff, and the school board, but planning 
to are 73.9 percent, 58.3 percent, and 65.8 percent, respectively. 

All regions made greater use of the report with administrative/ 
supervisory staff than with either teaching staff or school boards, as 
indicated in table 3. 



10 



TABLE J 

DISSEMINATION OP PERFORMANCE INDICATORS REPOIlT BY fitJPFBttfT»«np«.r 
TO A«INISTF^TIVE STAFF. TEACHINC%%a"'?'^NO^Sc25"S ' 
BY REGION AND TOTAL STATE 



Priicncid Co Adalnlicridv*/ 
SupTvltory Sciff 



rraiint*d to 
Tiichtng sciff 



FriavnCad co 
School Boird 



Ritlon 




¥aa 




No 


Fla 


n Co 


Yaa 


No 


[Plan Co 


Yaa 


No 


rian CO 




N 


I 


N 


X 


N 


X 


N 


X 


N 


X 


N 


X 


N 


1 X 


N 


' X 


H 


1 X 


C«plcal Diicrlcc Rittoa 


S 


71,4 


2 


28.6 


2 


28.6 


0 


0.0 


7 


100.0 


4 


57.1 




0.0 


7 


T — 
j 100.0 


4 




Buffalo Ration 


a 


BO.O 


2 


20, 0 


2 


20.0 


6 


60.0 


4 


40.0 


4 


40.0 




30.0 




1 

1 70.0 


5 


1 50.0 


Long I aland Ration 






5 


26. 3 


4 


21.1 


6 


31.6 


13 


68.4 


9 


47.4 




42.1 


■■' 




9 




Elatra Ritton 




61.8 


2 


18.2 


2 


18.2 


4 


36.4 


7 


hj.6 


5 


45.5 


3 


27.3 




j 72.7 


' 


63.h 


Syracuae Ration 


: 


100.0 


0 


0.0 


0 


0.0 


2 


33.3 


4 


66. 7 


' 


50.0 


1 


16.7 




1 83.3 


3 


50.0 


RechaaCar Rag Ion 




SO.O 




10.0 


0 


0.0 


5 


50.0 


5 


50.0 


2 


20.0 


3 


30.0 


7 


1 

1 70.0 


•i 


SO.O 


Hohawk Vallay Region 




S3. 3 




16. 7 


. 


lb. 7 






4 


66. 7 


3 


50.0 


3 


50.0 


3 


1 50.0 


3 


50.0 


Hld'HiKfaon Rag Ion 


7 


87.5 




... 


0 


0.0 






7 


87.5 


' 


3/.5 


' 




5 


i 62.5 


2 


2 5.0 


NorChtTD Raglon 




75.0 




25.0 




n.o 


7 


43.6 


9 


5b. J 


7 j 


43.8 


b 




10 


j 62.5 


7 


43.8 


Blnghncoo Raglon 




75.0 


2 \ 


25.0 




25.0 


3 


37.5 


5 


62.5 


^! 


25.0 


3 




5 


, 62. -i 


2 


25.0 


Rockland >Waatchaa car Raglon j 


i 

s 

_j 




3 1 


37.5, 




0.0^ 


1 


12.5 


7 ^ 


B7.5] 


0 ' 

— L 


0.0] 


3 




5 




1 


12. 5 


TOTAL STATE j 


66 j 


78.9 


73 1 


21.1 


"1 


— \ 

15. 6{ 


37 


33.9 


"i 


66. l| 


"i 


38. 5| 


"i 


1 

33.0! 


73 


h7.o| 


^8 


44.0 



NOTt: All p«rc«cai.i •» ^•Uulac.il bai.il cm dlicrlcci ricilvlni chr report. 



An effort was made to examine possible relationships between how the 
report was received (meeting or mail) and whether it was presented to 
local administrative and/or teaching staff and school boards. 

Three twofold tables were constructed and the data were tabulated 
according to how the report was received and whether it was presented to 
each of the three groups. Chi square tests using Yates' correction for 
continuity and one degree of freedom were applied. The results are 
shown in tables 4, 5, and 6. 

TABLE 4 

RELATIONSHIP BETWEEN HOW REPORT WAS RECEIVED 
AND WHETHER IT WAS REPORTED TO ADMINISTRATIVE STAFF 





How Received 




Meeting 


Mail 


Presented 


Yes 


55 


31 


No 


11 


12 






109 Total 



18 

11 



Simple observation indicates that districts receiving their report 
at a meeting were slightly more likely to share the results with their 
administrative staff; however, a value of 1,35 was obtained which is 
not significant at the .05 level. 



TABLE 5 

RELATIONSHIP BETWEEN HOW REPORT WAS RECEIVED 
AND WHETHER IT WAS REPORTED TO TEACHING STAFF 





How Received 




Meeting^ 


Mail 


Presented 


Yes 


25 


12 


No 


41 


31 




109 Total 



Again, observation indicates that districts receiving their report 
at a meeting were more likely to share the results with their staff; 
However, a X2 value of It 76 was obtained which is not significant at 
the .05 level. 



TABLE 6 

RELATIONSHIP BETWEEN HOW REPORT WAS RECEIVED 
AND WHETHER IT WAS REPORTED TO THE SCHOOL BOARD 





How Received 




Meef:ing 


Mall 


Presented 


Yes 


33 


3 


No 


33 


40 




109 Total 



A value of 19.88 was obtained which is significant at the 
.05 level. More reports were made to school boards when information was 
received at meetings than when it was received through the mail. 
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An additional comparison was made using the data collected for 
tables 1 and 12, to examine the relationship beti^een how the report was 
received and whether the respondent expressed a favorable, unfavorable, 
or neutral attitude toward the Performance Indicators program. 



TABLE 7 



RELATIONSHIP BETWEEN HOW REPORT WAS RECEIVED AND 
ATTITUDE TOWARD PERFORMANCE INDICATORS PROGRAM 





How Received 




Meeting 


Mail 


Favorable 


44 


20 


Unfavorable 


22 


23 






109 Total 



A value of 3.57 was computed which is not significant at the 
.05 level. Thus, although there is a slight positive relationship 
between receiving the report at a meeting and favorable attitudes, the 
test of significance indicates that it is not unlikely that the relation- 
ship could have occurred by chance. 

Respondents in 77.1 percent of the districts in the State receiving 
the report reported an agreement between Performance Indicators data and 
other information concerning their individual districts. The responses 
by region are presented in table 8. 
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Table b 



ACRECMENT OF PERPOKMAHCE INDICATORS QATA WITH 
OTHEK EVALUATION INTOiUUIION 
BY REGION AND TOTAL STATE 
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Syrscute Region 




100.0 


0 


0.0 


0 


0.0 


Rochetttr Region 




60.0 


0 


0.0 


4 


40.0 


Hohwak Valley Region 




8).) 


1 


16.7 


0 


0.0 


Htd-Hudtnn region 




75.0 


1 


12.5 


1 


12.5 


Northern Region 


11 


68.8 


3 


18.8 


2 


12.5 


Pinghaaton Region 




75.0 


1 


12.5 


1 


12.5 


Rockland-Ueitchettrr Region 




87,5 


0 


0.0 


1 


12.5 


TOTAL STATE 


84 


77.1 


9 


8.3 


16 


14.7 



NOTE: All pcrc*nt4gr> mre calculated bttcd on dittrictt indicating faBllitrity with 
the lt:purt. 



A large proportion (38.5%) of the respondents suggested that socio- 
economic data be included in the prediction of academic performance. Per- 
centages of respondents suggesting data in other categories were individual 
pupil data, 18-3 percent; student mobility, 13.8 percent; income of students' 
families, 11 percent; and proportion of students from families on public 
assistance, 10.1 percent. The "Other" category, which contained responses 
from 18.3 percent of the districts, included variables such as class size, 
program variables, parents' attitudes and educational background, teacher 
characteristics, affective measures, and per pupil expenditures. Sugges- 
tions that pupil ability or I.Q, scores be included were classified under 
"Individual Data." 

Results are summarized in table 9. 
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TABLE 9 



VAKUBLCS SUrcZSTES FOR IHaUSICH IN STUDIES TO 
rUSICT ACADEMIC FERPOWANCC 
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l'..3 


1 


U, 3 


0 


0, 0 


0 


0.0 




71, A 


Ittiffile RiKlon 
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CO 
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3 
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I 
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■ 2 
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2 
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0.0 


7 


18.2 
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0.0 


t. 


36. A 


2 


18.2 




^5.5 


Syrscuss Ragton 


1 


16.7 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


1 


16.7 


A 


66. 7 


Roclisttsr Region 


{, 


'•O.O 


0 


0.0 


2 


2U.U 




0.0 


2 


20.0 


I 


1 0.0 




50.0 


Hohawk Vallty Rexton 


2 


33.3 




16. 7 


0 


0.0 


2 


33.3 


I 


16. 7 


2 


3 3.3 




13.3 


Mld*4liidson Rtgton 


6 


75.0 


2 


2 5.0 


7 


25.0 




25.(1 


n 


0.0 


1 

0 1 


0.0 




23.0 


Morthatn Riglon 


U 


B7.5 


7 


A3. 8 




25.0 


2 1 


12.,; 


1 


6.3 


t 

0 1 


n.o 




6. 3 


Blnghmtm RsKtoa 


3 


37.5 


0 


0.0 


0 


0.0 


0 1 


o.oi 


= ! 


17.5 


1 

^ ! 


25.0 


1 1 


17.5 


Rocklind-ViatchsitiT RiRlon 


A 


50.0 


0 


0.0 


0 


0.0 


2 i 


25. o| 


' ! 


62.5| 


^ 1 


87.51 


•i 


0.0 


I 

TOTAL STATE 1 

1 


'•2 


38.5 


U 


10. 1 




II. 0 


>5 1 


— r 

13.8| 


20 j 


1 

18. 3^ 

1 


20 ; 


18. 3| 

1 


j 3 7.6 



■ultipl. re.ponie.. percent, do not Cocil lOOl. i«mm.rlty 



^ith the rrport. Due 



lSocio«cono«lc icaCul of •cudcnti or f.ailles 



in the dlitrict . 



Proportion of itudent. fro. f«ili., on public ...i.t.nce. 

Of the districts familiar with the report, fewer than half made 
suggestions about other criteria of school quality which might be used 
in developing Performance Indicators, The largest single category of 
responses were classified as "Other." This category included such 



TABU 10 

SUCCESTH) CRITERU OF SCHOOL QUALITY 



Region 



Caplfal Oiitrlct Riglon 
Buffalo Rgglon 
Lonjt Iilind Rsglon 
Elatri Ragion 
S^rricuig Region 
Rochigtir Rigicfl 
Mohawk ValUy Ragton 
Mtd-Htidgon Rag 1 mi 
Nttrtharn Rigton 
Binghnton Riglon 
RocklindoWiatrhiatir Ragton 



TOTAL STATE 



Other 
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0.01 

lo.oj 

o.oj 

o.o! 

t 

10. o| 

n,o' 
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U.5^ 

o.oj 



1 I 
I : 



o.oj n 

lo.nt b 

5. 1^ : 

9.l' 1 

I 

iii.r| 0 

16. 7| [> 

2 5.o' 0 

\K6\ 3 

12.5] 0 

12.5' 0 

I 



0.0 0 

o.oj 0 

10.5! 0 

<».l{ 2 

D.c! 0 

II. oj 0 

ft.n^ I) 

0.0; 1 

18, 8 J I 

O.oj 0 

0,0* 0 



7.3; 12 I I I.e.' 6 



0.0; 0 
0. n" I 



1 



0.0 

O.fjt I 

o.oj f, 

12,5' ! 

6. 3| 2 

o.n' p 

o.o' n 



0.0 0 

I 

0.0 

l\ 1 
lt>. 7^ 
10.()| 
0.0' 



1.7, H B. 3 2 



:2.^ 

t 

12.5,' 

o.o' 



NOTE! All pirctnt«g«a ir* calculated b«ied ondiatrlctt tndf.t.njt f.nUt«rtty wl;'i .v r^P. 
njltlpla retpon»«l, percenta do not total 1001. 



0.0 
10. {) 
36.8; 

9. 1] 
66. 7] 
50. oj 
t>6. 7! 
12.5! 
18.8]' 
12.5' 
2,.c' 



I 100.0 
j 70.0 

I 

! 

I 3C.0 
I 16.7 

I 3,., 

I 50 0 
I 75.0 
I 62.5 



] 26.6; I 50.5 
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suggestions as curriculum indicators other than reading and mathematics, 
affective measures, and teacher effectiveness. The low response rate 
suggests that this item requires a more considered response than is 
possible on a telephone survey. Table 10 presents the results about 
suggested criteria of school quality. 

Only 6.4 percent of districts in the State receiving the Performance 
Indicators report preferred to calculate their own results if equations, 
worksheets, and profiles were provided. Recurring responses indicated a 
perceived lack of personnel and competencies necessary to carry out the 
statistical analyses. Other responses indicated a desire for the State 
Education Department to continue Performance Indicators reports. 

TABLE 1 1 

NlUhER AND PERCEKf CF DISTRICTS 
rK£rERKIMC TO CALCtXUE OWN RfiSlO-TS 



Rf XlOD 


Would Prefer To CnlcutAtr 
(Vn RcHulti from Equailont 
and WorkltirrtA 








C«pltAl District Rj^Klon 


i 1 

2 


26.6 


RuffAlo R^Kton 


0 


0,0 


UtRK iHlAnd Region 


0 


0,0 


ElHtra Region 


I 


I 


SyrAcu«# Rritlon 


1 


16, 7 


1tM;hrit.cr RtkIdh 




I't.O 


HotiAwk Vallry Rck''"^ 


c 


O.G 


Hid-Hudit>n Rj-Klon 


■ 




Northern R*;|{lmi 




6.] 


f>liigh«aton K«Klon 




0,0 


RockUnd'Uritchrit^r Rfglon 






0,0 


rOTAL STATE 


1 



NOTFt All prrcrnCnAr* cjlcuUtrd b«ie<l ondtaCrtcti Indicating frattlarlCy 
ultti t.hr report . 

Respondents in 58.7 percent of the districts expressing familiarity 
with the Performance Indi*_.^tors report were Identified by the callers as 
making favorable comments about the report. Two percent stated unfavorable 
attitudes. 
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TABLE 12 

ATHWOE TOWARDS THE PERFORMANCE INDICATORS PROJECT 





Pavo 




Unfavorable 


Neut ral 


or Hixad 


N 


X 


H 


X 


N 


X 


C«piCAl Dlicricc Xasion 


4 




57. IX 


0 


o.ox 


3 


42. 9X 


Buffalo X«gion 


10 
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0 


0.0 


0 


0.0 


Umil ItlAnd Ration 






47.4 


0 


0, 0 


10 


S2.6 
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0 


o.c 


: 


18.2 
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SO. 0 


0 


0.0 




50.0 


SloclMiC«r Ragion 






so.o 


0 


0.0 




50.0 
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so.o 


0 


0.0 




50.0 
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37.5 


1 


12.3 




SO.O 


Northcnt Ragion 


12 




75.0 


0 


0.0 




25.0 


BinghaaCon Ragion 






50.0 


0 


0.0 




50.0 


RoekUi>d*Wettch«gt«r R^gton 






23. 'O 


1 


12.5 




62.5 


TOTAL STATE 


U 


58. 7 

1 


2 


i.e 


43 


39.4 



HOTEJ All p«rc«nt«gai ar« c«1luUc»41 h*fd on dlttrlcti indlciclnd f«Biliarlty wich the report. 

Callers Evaluation 

The four callers for the Performance Indicators in Education 
Telephone Survey completed a one-page questionnaire giving their own 
evaluations of the survey. Following are the results of the evaluation. 

All four callers indicated that the respondents' general attitudes 
toward the survey were "pleasant and agreeable." This implies that in 
using the telephone survey technique, good rapport can be established with 
the respondents and that the survey can be successfully completed. 

The callers wore asked to describe any unusual response to the 
questions of the survey. Recurring comments are summarized below: 

--Several respondents expressed an uneasiness about possible 
adverse effects of making the report available to the news 
media • 

--The validity of the Pupil Evaluation Program tests as an 
acceptable criteria for school performance was questioned, 
especially in terms of making comparisons among schools. 
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There seemed to be a failure on the part of some respondents 
to recognize that the Performance Indicators program is 
an attempt to develop more defensible ways of measuring 
school performance by statistically controlling for 
certain input variables. 
--Several respondents recommended cleemphasizing cognitive 
measures with more emphasis on the affective domain. 
Others suggested broadened criteria of school success to 
include cognitive areas besides just reading and mathematics. 
—The time lag between testing (1968, 1969, 1970) and actual 

dissemination of reports (1972) was cited as a disadvantage. 
— A general consensus was registered for more cooperation 
between the Department and districts in the development 
of accountability models. 
Callers were asked on questior* 3, '*Would a training session or 
trial run for callers be worthwhile?*' Mixed feelings were recorded on 
this and question 4, which asked for suggestions for improving the survey 
process in the future. As discussed above an attempt was made to select 
callers with limited familiarity with the project to avoid biasing 
results and co speed interviews. The callers felt, however, that this 
limited knowledge of PIE, as well as the surprise nature of the calls, 
may have prevented the recording of all the information available. 
Several alternatives were suggested: 

1. Structure the survey to exclude multiple category 
questions like numbers 7 and 8. 
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2. Conduct training sessions, not for practice in making 
calls, but to give the interviewer more background 
about the program in question. 

3. Notify prospective respondents in advance so they can 
prepare for the interview. 

In summary, the callers felt they had good success in conducting 
the survey. The respondents were "pleasant and agreeable" even though 
some were unfamiliar with the program. More preparation for both the 
callers and respondents was recommended. 
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CHAPTER IV. 
CONCLUSIONS AND RECOMMENDATIONS 



Telephone Survey , The telephone survey was demonstrated to have 
unsuspected potential when applied to education. The response rate 
equaled or exceeded that commonly found in published research using 
mailed surveys. It appears that while the methodological considerations 
are complex, they are not insurmountable. Continued efforts should be 
made to discover ways of overcoming these difficulties. The conceptual 
model proved realistic ia application. 

It is recommendeJ that educators learn to avail themselves of this 
and other datci-collecting devices which have been used so widely and 
effectively by induscry., marketing, and political groups. A careful 
analysis should be made of the types of information which can be 
effectively obtained through the telephone survey technique. Information 
which needs to be thought through ahead of time or which requires careful 
organization might be collected better through printed questionnaires • 

Performance Indicators in Education . A high percent of the respondents 
reported unf amiliarity with Performance Indicators. This was surprising 
since Bureau of School Programs Evaluation records show that the reports 
were disseminated to all 140 districts. In general it can be stated 
that more reports were shared with administrative staffs than with 
teachers or school boards, that data reported to districts agreed with 
other information, that most districts preferred not to calculate their 
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own results, and that attitudes toward the Performance Indicators program 
were favorable. 

I 

The lack of positive results concerning the relationship between 
how the report was received (at a meeting or by mail) suggests that 
results of future meetings should be vigorously evaluated to determine 
their effectiveness. 

Recommendations for improving Performance Indicators suggest that 
including socioeconomic status, affective measures, and program variables 
would make the reports more acceptable and useful to the districts. The 
validity of the Pupil Evaluation Program tests as a criterion was 
questioned, but few respondents were able to suggest alternatives. 

Uneasiness expressed about possible adverse effects of making the 
report public suggests that the program should be continued on an 
experimental basis at this time. 
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APPENDIX A 



PERFORMANCE INDICATORS TELEPHONE SURVEY 

Callers' Instructions and Guide 12/18/72 

Monday 

General Instruction s 

Please maintain a friendly but businesslike manner throughout the 
survey. Refrain from commenting about the views of the person you are 
talking to or those of earlier interviewees. Rather than answering specific 
questions about the Performance Indicators program, be prepared to point 
out that you do not work in the program but that we ave interested in getting 
his views so we can improve the program. 

If you get a busy signal or no answer after 10 rings, terniinate and 
go on to the next number. Try again later and periodically as you go through 
your list. 

Responses of the interviewees are to be recorded on the Call Record 
Sheet. It is designed so that most responses you get can be recorded with a 
check mark. Item 15 is an exception. You may find also that some responses 
do not fit into any of the predetermined categories. In those cases, try to 
capture the main idea in as few words as possible. 

We would like to get information from the superintendent (or supervising 
principal). If he refers you to someone else when he learns the r.,.aa:e of the 
call, be sure to record the new person's name, title, and telephone number. 
But do not interview someone else simply because the superintendent is not 
available at the time. Call back later. 



Guide for Calling 

When the telephone is answered say: "My name is 

I'm calling from the State Education Department. May I speak to Mr. ? 

When the superintendent answers say: "Hello. My name is 

I'm calling from the State Education Department. We are conducting a survey of 
districts which received Performance Indicators reports, avd I would like to ask 
you some questions." 

(If superintendent indicates that he is not familiar with the Performance 
Indicarors report, ask if a member of his staff could answer questions about 
it. Get the person's name and telephone number, if no one in the district 
knows anything about the Performance Indicators report say, "Would you like 
to receive some information about it? We'll send you a copy of a publication 
describing the program. Thank you. Good-by.") 

1. Did you receive a Performance Indicators in Education report for your 
district? (If No, ask, "Would you like a description of the project 
mailed to you?" and terminate.) 

2. Was it received at a meeting or by mail? 
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3. Has the report been presented to administrative/advisory staff? (If 
No, ask, "Do you have plans to present it to them?") 

4. Has the report been presented to the teaching staff. (If No, ask, 
"Do you have plans to present it to them?") 

5. Has the report been presented to the school board? (If No, ask, 
"Do you have plans to present it to them?") 

6. Do the results reported to you generally agree with other information 
you have about your district? 

7. What other variables do you think should be included in the prediction af 
academi c performance? 

8. What other criteria of school quality do you think should be considered? 

9. Would you prefer to calculate your own results if equations, worksheets, 
and profiles wore provided rather than having the completed report sent 
to you? 

10. Do you have other comments which you would like to make about the 
Performance Indicators program? (Tone of remarks is generally: 

favorable 

unfavorable 

neutral or mixed 

(Try to jot down in a few words the kinds of criticisms or suggestions 
the interviewee makes.) 



After you have finished your list, recall all busy and no answer 
numbers. If you receive a busy or no answer again, then terminate. 

PI easfi total all columns on your Call Record Sheet and turn it in. 
When you have done this, you have finished your assignment. We will let you 
know whether you can help later in calling those numbers you weren't able to 
reach. 

PLEASE COMPLETE THE ATTACHED CALLER'S EVALUATION. 



Thank you for your important contribution to this effort. 
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APPENDIX B 



PERFORMANCE INDICATORS TELEPHONE SURVEY 
CALL RECORD SHEET 
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APPENDIX C 
PERFORMANCE INDICATORS TELEPHONE SURVEY 
CALLER'S EVALUATION 



Check your general Impression of the respondees' attitude toward 
the survey. (Check one which best typifies all contacts.) 



_Pleasant and agreeable 
Lukewa rra 



.Hostile, di3interested,and/or disagreeable 



Describe any unusual response to: 

Questions 3.4^5 (Has the report been presented to 

supervisors, teachers, or board members?) 



Question 9 (Would you prefer to calculate your own results?) 
Question 10 (Do you have other comments?) 

Would a training session or trial run for callers be worthwhile? 
Yes No 



What suggestions do you have for improving the survey process 
in the future? 
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